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TITLE OF THE INVENTION 



INFORMATION PROCESSING APPARATUS , MIXED REALITY 



PRESENTATION APPARATUS, METHOD THEREOF, AND STORAGE 



MEDIUM 



5 



FIELD OF THE INVENTION 



The present invention relates to an information 
processing apparatus that derives the calibration 
information needed to measure the position and/or 

10 attitude of a measuring object based on the output 

values of a position and/or attitude sensor, a mixed 
reality presentation apparatus that displays virtual 
space superimposed over a picture of captured real 
space on a display screen or displays virtual space 

15 superimposed over the real space transmitted optically 
through a display screen, a method thereof, and a 
storage medium. 



mixed reality (hereinafter abbreviated to MR). MR aims 
at seamless connection between real space and virtual 
space. It is an important technique in virtual reality 
(hereinafter abbreviated to VR) . 



world which can conventionally be experienced only in 



BACKGROUND OF THE INVENTION 



20 



Recently, studies have been conducted actively on 



25 



MR aims at coexistence of real space and a VR 
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situations isolated from real space. It attracts 
attention as a technique for enhancing VR. 



10 



MR includes video see- through mode that involves 
superimposing a picture of virtual space (for example, 
virtual objects dreiWi by computer graphics (hereinafter 
abbreviated to CG) or\ext information) over a picture 

of captured real space ^kpjtured by an imaging apparatus 

Nil 

such as a video camera and optical see -through mode 



that involves superimposing a picture of virtual space 



over a picture of the real space transmitted optically 



through a display screen , 





MET"; IlTlixpe^^^ 
aids which will show the inside of the patient's body 
to the surgeon as if it were seen through, applications 

15 in work aids which will show assembly procedures of a 
product superimposed over the actual objects in a 
factory, and other applications totalfe^dif f erent in 

quality from those of conventional VR. 



What is commonly required of these applications is 
20 a technique for aligning the position/attitude of real 
space and virtual space and many attempts have been 
made so far. 

The problem of position/attitude alignment in 
video see -through mode MR resolves itself down to a 
25 matter of finding the 3D position/attitude of a video 
camera in a global coordinate system set up in real 
space (hereinafter referred to simply as a global 
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coordinate system) . Similarly, the problem of 
position/attitude alignment in optical see-through mode 
MR comes down to a matter of finding the 3D 
position/attitude of the user's viewpoint. 
5 As a method of solving these problems , it is 

common practice to derive the 3D position/attitude of 
the video camera's or user's viewpoint in a global 
coordinate system by using a 3D position/attitude 
sensor such as a magnetic sensor or ultrasonic sensor. 
10 The output values of a sensor represent, in the 

case of a magnetic sensor, for example, the 3D 
position/attitude of the sensor itself in the sensor 
coordinate system whose center is defined as the source 
of an AC magnetic field described later, and not the 3D 
15 position/attitude of the video camera's or user's 

viewpoint in a global coordinate system being measured. 
In other words, the output values of the sensor cannot 
be used directly as the 3D position/attitude of the 
video camera's or user's viewpoint, but some coordinate 
20 transformation is necessary. Hereinafter the data 

needed for coordinate transformation between the output 
values of the sensor and the 3D position/attitude of 
the video camera's or user's viewpoint will be referred 
to as calibration data (calibration information). Also, 
25 the process of setting or calculating calibration data 
beforehand will be referred to as calibration. Only 
after accurate calibration is performed, the output of 
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a 3D position/attitude sensor can be converted to the 
3D position/attitude of the video camera's or user's 
viewpoint and accurate position/attitude alignment in 
MR can be performed, 
5 Conventionally, however, it is difficult to: 

• acquire the position/attitude (the information 
needed to convert sensor output to the 3D 
position/attitude of the sensor in a global coordinate 
system) of the sensor coordinate system in a global 

10 coordinate system, and 

* acquire the position/attitude (the information 
needed to transform the position/attitude of the sensor 
into the position/attitude of the measuring object) of 
the measuring object (camera's or user's viewpoint) as 

15 viewed from the sensor. 

The present invention has been made in view of the 
above problems. Its object is to acquire the 
information needed to transform a sensor coordinate 
system into a global coordinate system and the 

20 information needed to transform the position/attitude 
of a sensor into the position/attitude of a measuring 
object . 



SUMMARY OF THE INVENTION 
2 5 To achieve the object of the present invention, an 

information processing apparatus according to the 
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present invention, for example, has the following 
configuration . 

Specifically, it is an information processing 
apparatus that derives the calibration information 
5 needed to measure the position and/or attitude of a 
measuring object based on the output values of a 
position and/or attitude sensor, comprising: 

input means for entering information about a match 
between the position and/or attitude of the above 
10 described measuring object and a predetermined position 
and/or attitude; 

acquisition means for acquiring the output values 
from the above described position and/or attitude 
sensor according to the input by the above described 
15 input means; and 

operation means for deriving the above described 
calibration information, based on the above described 
predetermined position and/or attitude and the output 
values of the above described position and/or attitude 
20 sensor acquired by the above described acquisition 
means . 

To achieve the object of the present invention, a 
mixed reality presentation apparatus according to the 
present invention, for example, has the following 
25 configuration . 

Specifically, it is a mixed reality presentation 
apparatus that displays virtual space superimposed over 
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a picture of captured real space on a display screen or 
displays virtual space superimposed over the real space 
transmitted optically through a display screen based on 
output values of a position and/or attitude sensor, 
5 comprising : 

an information processing apparatus recited in any 
of claims 10 to 16; and 

switching means for switching between presentation 
mode that presents mixed reality and derivation mode 
10 that derives calibration information; wherein 
the calibration information needed for 
presentation of the above described mixed reality is 
derived in the above described derivation mode and 
mixed reality is presented using the calibration 
15 information in the above described presentation mode. 
Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
20 the same or similar parts throughout the figures 
thereof . 

BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings , which are incorporated 
25 in and constitute a part of the specification, 

illustrate embodiments of the invention and, together 
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with the description, serve to explain the principles 
of the invention . 

FIG. 1 is a side view of the game device of the 
system according to a first embodiment of the present 
5 invention; 

FIG. 2 is a drawing showing a picture as viewed by 
left player 2000 through his/her own HMD 210L; 

FIG. 3 is a drawing showing the configuration of 
HMD 210; 

10 FIG. 4 is a drawing illustrating an outline of 

calibration setup ; 

FIG. 5 is a drawing showing detailed configuration 

of the calibration section 5055L according to the first 

embodiment of the present invention; 
15 FIG. 6 is a process flowchart of the calibration 

apparatus according to the first embodiment of the 

present invention ; 

FIG. 7A is a drawing showing the image displayed 

on a display section 210L in Step S1040; 
20 FIG. 7B is a drawing showing a picture optically 

transmitted through the display section 210L and viewed 

by a player; 

FIG. 7C is a drawing showing how the positions of 
a virtual marker picture and a real marker picture are 
2 5 superimposed exactly by parallel translation or 

rotation of HMD 210L (i.e., viewpoint 1901) in Step 
S1070; 



- 7 - 



FIG. 8 is a drawing showing the configuration of a 
3D image generation/presentation system that utilizes 
the calibration data obtained through calibration on 
the game device of FIG. 1; 
5 FIG. 9 is a flowchart showing the control 

procedures for measurement of mallet position; 

FIG. 10 is a flowchart showing the control 
procedures for measurement of mallet position; 

FIG. 11 is a flowchart showing the control 
10 procedures for measurement of mallet position; 

FIG. 12 is a drawing illustrating the split halves 
of the image obtained by a CCD camera 230 located in a 
fixed position; 

FIG. 13 is a drawing showing a search area; 
15 FIG. 14 is a drawing showing the game field for 

air hockey games according to the first embodiment of 
the present invention; 

FIG. 15 is a flowchart illustrating the processing 
procedures carried out in a game status control section 
20 5030; 

FIG. 16 is a drawing illustrating the definitions 
of width and height ; 

FIG. 17 is a flowchart of processes carried out in 
position/attitude transformation sections 5060 (5060L, 
25 5060R) and image generation sections 5050 (5050L, 
5050R) ; 
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FIG. 18 is a conceptual diagram illustrating how a 
magnetic sensor measures its position/attitude; 

FIG. 19 is a drawing illustrating how the 
position/attitude transformation section 5060L 
determines the position/attitude Mvw of the viewpoint 
1901 of the player 2000 in a global coordinate system 
8000; and 

FIG. 20 is a drawing showing the equipment 
configuration used for the calibration of the game 
device according to the first embodiment of the present 
invention . 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
accompanying drawings . 
[ Firs t Embodiment ] 

Now detailed description will be given about cases 
in which the information processing apparatus and 
method thereof according to the present invention are 
applied to the calibration apparatus and method thereof 
for the calibration of a game device, and furthermore, 
for example, to an air hockey game device that utilizes 
MR technology. 

The air hockey game is a game which is played 
between two opposing players who try to score goals by 
shooting a puck floated by compressed air supplied from 



. o o 

below, into the opponent's cage. The one who scores 
more goals wins . In the air hockey game that uses the 
MR technology of this embodiment, the puck is presented 
as a virtual 3D image superimposed over a table in the 
5 real environment to the players , who compete to hit it 
with real mallets . 

<Conf iguration of the game device) 

FIG. 1 is a side view of the game device of the 
system according to this embodiment. In the air hockey 

10 game that uses the MR technology, two players 2000 and 
3000 face each other across a table 1000 with a mallet 
(260L or 260R) in their hand. The two players 2000 and 
3000 wear head-mounted displays 210L and 210R 
(hereinafter abbreviated to HMD), respectively, on 

15 their head. The mallets (260L and 260 R) of this 

embodiment have an infrared emitter at their tip. The 
infrared rays emitted from the infrared emitter are 
used to detect the positions of the mallets. Details 
will be described later. Incidentally, if the mallets 

20 have distinctive shapes or colors, their positions may 
be detected through pattern recognition using these 
distinctive features . 

The HMDs 210L and 210R of this embodiment are a 
see-through type as shown in FIG. 3. They will be 

25 described in detail later. The two players 2000 and 

3000 can see the surface of the table even if they wear 
the HMD 210L or 210R. Also, the HMDs 210L and 210R 



- 10 - 



0 

O 



O 



receive 3D virtual images from an image processing 
system described later. Therefore, the players 2 000 
and 3000 view a 3D virtual picture displayed on the 
display screen of the HMD 210L or 210R and superimposed 
5 over the real space transmitted through the optical 
system (not shown in FIG. 1) of the HMD 210L or 210R 

FIG. 2 is a drawing showing a picture as viewed by 
the left player 2000 through his/her own HMD 210L. The 
two players compete to shoot a puck 1500 represented by 

10 a virtual picture. The player 2000 uses the real 

mallet 260L held in his/her hand to hit the puck. The 
goal 1200R is visible in front of the opposition player 
3000. The image processing system (not shown in FIG. 
2) generates a three-dimensional CG and displays it on 

15 the HMD 210L so that the goal 1200R will be visible 
near the opponent . 

The opposition player 3000 will also see a virtual 
goal (not shown) near the player 2000 through the HMD 
210R. The puck 1500 is also generated by the image 

20 processing system (not shown) and displayed on each HMD 
<HMD with a magnetic sensor > 

FIG. 3 shows the configuration of the HMD 210. 
This HMD 210 is the see- through type. It is, for 
example the HMD body disclosed by Japanese published 

25 unexamined application H7-333551 with a magnetic sensor 
220 mounted via a support rod 221. Reference numeral 
211 denotes an LCD panel. The light (the picture 
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presented to the observer) from the LCD panel 211 
enters an optical member 212, reflects off a totally 
reflective surface 214, reflects off the totally 
reflective surface of a concave mirror 213, and reaches 
5 the observer's eyes passing through the totally 
reflective surface 214. 

The magnetic sensor 220 measures the 
position/attitude of the observer's viewpoint. Since 
the magnetic sensor is vulnerable to magnetic noise, it 
10 is separated from the LCD panel 211, which is a noise 
source, by the support rod 221. 

Incidentally, the configuration in which a 
magnetic sensor 220 is mounted on an HMD 210 as shown 
in FIG. 3 is not limited to optical see- through type 
15 HMDs. It may also be applied to video see-through type 
HMDs of for the purpose of detecting the 
position/attitude of the video camera's viewpoint. 

In FIG. 1, the HMDs 210L and 210R are secured to 
the head of the players 2000 and 3000, respectively, 
20 with a band (not shown). The players have the magnetic 
sensor 220 (220L or 220R) shown in FIGS. 1 and 3 as 
well as a CCD camera 240 (240L or 240R) , imaging means, 
fastened to their head. The field of view of the CCD 
camera 240 is set to the area in front of the player. 
25 In the case of air hockey games, since the players look 
at the top surface of the table 1000, the cameras 240 
also capture the surface of the table 1000. An AC 
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magnetic field generator 250 is fixed at any such place 
that the magnetic sensors 220 fall within its range. 
The magnetic sensors 220 sense the changes in the AC 
magnetic field generated by the AC magnetic field 
5 generator 250. 

According to the changes in the AC magnetic field 
sensed by the magnetic sensors 220, a position/attitude 
measuring section 5000 (not shown in FIG. 1) measures 
the positions/attitudes of the magnetic sensors 220 in 

10 the sensor coordinate system 8010 defined around the AC 
magnetic field generator 250. Position/attitude 
transformation sections 5060 (5060L and 5060R) (not 
shown in FIG. 1) transform the positions/attitudes of 
the magnetic sensors 220 in the sensor coordinate 

15 system 8010 into the positions/attitudes of the 

players 1 viewpoints in the global coordinate system 
8000, based on the calibration data (acquired by a 
calibration technique described later) stored in 
calibration data storage sections 5080 (5080L and 

20 5080R) (not shown in FIG. 1). Incidentally, as the 

magnetic sensors 220, AC magnetic field generator 250, 
and position/attitude measuring sections 5000, this 
embodiment employs position/attitude measuring system 
Fastrak from Polhemus Inc. Besides, the 

25 position/attitude transformation sections 5060 and 

calibration data storage sections 5080 are implemented, 
for example, using an Onyx2 computer system from SGI. 
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When the player lowers his/her gaze obliquely to 
look at the surface of the table 1000, the changes in 
the position/attitude of the player's viewpoint are 
detected by the magnetic sensor 220. The surface of 
5 the table 1000, virtual puck 1500 described above, real 
mallets 260 (260L and 260 R) , and virtual goals 1200 
(1200L and 1200R) are shown through the HMD 210, based 
on the detected changes in the position/attitude. In 
this way, the changes in the position/attitude of the 

10 player's viewpoint is always detected by the magnetic 
sensor 220 and the player can see the picture and real 
space, based on the detected position/attitude. 

The position and attitude of an object B in a 
coordinate system A will be represented herein by a 4 x 

15 4 matrix M BA . In other words, it is the matrix for 

coordinate transformation from the coordinate system A 
into the coordinate system B defined by the object B 
and defines a transformation P B = M BA P A for changing 
coordinates P A = (X A , Y A , Z A , 1) T in the coordinate 

20 system A into coordinates P B = (X B , Y B , Z B , 1) T in the 
coordinate system B. To put it another way, the 
position/attitude Mvw of the viewpoint 1901 of the 
player 2000 in the global coordinate system 8000 is the 
coordinate transformation matrix (P v = Mvw Pw) for 

25 changing coordinates P w = (X w , Y„, Z w , 1) T in the global 

coordinate system 8000 into coordinates P v = (X v , Y v , Z v , 
1) T in the coordinate system of the player's viewpoint. 
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The transformation matrix M that represents the 
position/attitude of the object is given by the product 
of a rotation matrix R x around the X axis, rotation 
matrix R Y around the Y axis, rotation matrix R 2 around 
the Z axis, and translation matrix T (all are 4x4). 
The relationship M = RT = R z R x R Y T holds among these 
matrices, which are give by: 
[Equation 1] 



(x, y, z) = 

[Equation 2] 
Rx(6x) = 

[Equation 3] 



ri o o - x 

0 1 0 - y 

0 0 1 - z 

0 0 0 1 



10 0 0 

0 cos 0x sin 9x 0 

0 - sin 8x cos 9x 0 

0 0 0 1 



cos 9y 0 - sin 8y 



RY(6y) = 

[Equation 4 ] 
R2(9z) = 



0 1 
sin 0y 0 
0 0 



0 

cos 9y 
0 



cos 9z sin 8z 0 0 

- sin 0z cos 0z 0 0 

0 0 10 

0 0 0 1 



where x, y, z, 0x, 8y, and 0z represent the 
position/attitude of the object. More particularly, x, 
y, z represent the position, 0x the elevation angle, 9y 
the direction angle, and 9z the slope. 
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The following describes with reference to FIG. 19 
how the position/attitude transformation section 5060L 
v determines the position/attitude Mvw of the viewpoint 

1901 of the player 2000 in the global coordinate system 
5 8000. 

In FIG. 19, the position/attitude of the magnetic 
field generator 250 in the global coordinate system 
8000 (equivalent to the position/attitude of the sensor 
coordinate system 8010 in the global coordinate system 
10 8000) is denoted by M™, position/attitude of the 

magnetic sensor 220L in the sensor coordinate system 
8010 is denoted by M ST , and relative position/attitude 
of the viewpoint 1901 of the player 2000 as viewed from 
the magnetic sensor 220L is denoted by M V s- 
15 Then the position/attitude Mvw of the viewpoint 

1901 of the player 2000 in the global coordinate system 
8000 is given by: 

Mvw = M V s * M S t * Mtw (Equation A) 

Of these positions/attitudes, M ST is the input into 
20 the position/attitude transformation section 5060L, Mvw 
is the output from the position/attitude transformation 
section 5060L, and M V s and Mtw are the calibration data 
needed to convert M ST into Mvw- That is, the 
position/attitude transformation section 5060L 
25 calculates Mvw according to Equation A using the input 
M S t from the position/attitude measuring section 5000 
as well as M vs and Mtw stored in the calibration data 
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storage section 5080L described later and outputs it to 
the image generation section 5050L described later. 
<Conf iguration of equipment during calibration> 

FIG. 20 shows the equipment configuration used for 
5 the calibration of the game device according to this 
embodiment. As shown in FIG. 20, the calibration uses 
almost the same equipment configuration as that for 
game playing. Specifically, the configuration includes 
the AC magnetic field generator 250, magnetic sensor 
10 220L, position/attitude measuring section 5000, and HMD 
210L as is the case with the configuration for game 
playing, but it has a calibration section 5055L and 
operating section 5090 instead of the position/attitude 
transformation section 5060L and the image generation 
15 section 5050L. 

FIG. 4 shows an outline of calibration setup. To 
perform calibration, the position/attitude Mvw of the 
viewpoint 1901 of the player wearing the HMD 210L is 
moved to a predetermined position/attitude M°vw (or the 
20 HMD 210L is moved by an operator (not shown) to such a 
position/attitude that the viewpoint 1901 of the player 
will be placed in the position/attitude M°vw when the 
player puts on the HMD 210L) and the output M° ST from 
the position/attitude measuring section 5000 is 
25 acquired at that time. Using the operating section 
5090L, the player or unshown operator enters the 
information that the viewpoint 1901 is placed in the 
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position/attitude M°vw- According to the input into the 
operating section 5090L, the calibration section 5055L 
receives the output M° ST from the position/attitude 
measuring section 5000 at that time (i.e., at the time 
5 when the viewpoint 1901 is placed in the 

position/attitude M°vw) , calculates the calibration data 
M V s or Mtw, whichever is unknown, based on the 
position/attitude M°vw and position/attitude M° ST , and 
outputs the results to the calibration data storage 

10 section 5080L. 

With this embodiment, it is assumed that the 
matrix M-™ for coordinate transformation from the 
global coordinate system into the sensor coordinate 
system has been derived by some technique (e.g., a 

15 method described later) and that unknown calibration 

data is only M V s- The calibration data storage section 
5080L stores the calibration data M V s received from the 
calibration section 5055L and stores the Mtw as known 
values. The transformation matrices M V s and Mtw are 

20 referenced by the position/attitude transformation 
section 5060L during game playing. 

A calibration grid 6000 can be placed at a fixed 
position in real space and has on itself an established 
position that will serve as a guide when moving the HMD 

25 210L. The fixed position and established position have 
been designed such that when the calibration grid 6000 
is placed at the fixed position and the HMD 210L is 
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moved to the established position, the viewpoint 1901 
of the player who wears the HMD 210L will be placed 
near the position/attitude M°vw- The calibration grid 
6000 is built of plastic pipes to prevent interference 
5 when the magnetic sensor 220L receives the AC magnetic 
field outputted by the AC magnetic field generator 250 
and helps the viewpoint 1901 remain in the 
position/attitude M°vw. 

Markers 1600 (1600a, 1600b, ...) that have known 

10 values in the global coordinate system are placed on 

the table 1000. They are used in the processing by the 
calibration section 5055L (described later) to help the 
viewpoint 1901 remain in the position/attitude M°vw- 
Incidentally, this embodiment uses four markers 1600, 

15 but more markers may be used. 

Now the processing in the calibration section 
5055L will be described in detail. FIG. 5 shows 
detailed configuration of the calibration section 5055L 
according to this embodiment. The calibration section 

20 5055L is implemented, for example, using an Onyx2 
computer system. 

As shown in FIG. 5, the calibration section 5055L 
is composed of a memory 5045L, virtual marker picture 
generation section 5075L, marker location storage 

25 section 5035L, and calibration data calculation section 
5040L. 
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The marker location storage section 5035L stores 
the locations of the markers 1600 in the global 
coordinate system and outputs the marker locations to 
the virtual marker picture generation section 5075L. 

The memory 5045L stores the predetermined 
position/attitude M°vw of the viewpoint 1901. 

Based on the marker locations stored in the marker 
location storage section 5035L and the 

position/attitude M°vw of the viewpoint 1901 stored in 
the memory 5045L, the virtual marker picture generation 
section 5075L calculates the location of each marker on 
the display screen where the marker will be seen by the 
player if the viewpoint 1901 is in the 
position/attitude M°vw, generates a virtual picture 
representing an X mark at that location, and makes it 
displayed on the display section 210L. 

The display section 210L displays the virtual 
pictures of the markers generated by the virtual marker 
picture generation section 5075L while transmitting the 
markers 1600 in real space optically through itself. 
If the viewpoint 1901 is located in the 

position/attitude M°vw# the marker images in real space 
and the virtual pictures of the markers as viewed by 
the player should match. 

The operating section 5090L detects that the 
viewpoint 1901 is placed in the position/attitude M°vw, 
based on the information entered by the player or 
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unshown operator as described above, and sends an 
instruction to calculate calibration data to the 
calibration data calculation section 5040L. By 
rotation and translation of HMD 210L (and thus the 
5 viewpoint 1901), the player superimposes the virtual 
pictures of the markers displayed in the display 
section 210L and the marker images in the real space 
optically transmitted through the display section 210L, 
and when they are overlaid sufficiently, he/she enters 

10 input through the operating section 5090L, for example, 
by pressing a specific key. 

According to the instructions sent from the 
operating section 5090L, the calibration data 
calculation section 5040L receives the output M° ST from 

15 the position/attitude measuring section 5000 at the 
time when the instructions are entered (i.e., at the 
time when the viewpoint 1901 is placed in the 
position/attitude M°vw) as well as the position/attitude 
M°vw stored in the memory 504 5L and Mtw stored in the 

20 calibration data storage section 5080L. 

From Equation A, it follows that the following 
relationship exists among the data processed in the 
calibration data calculation section 5040L: 

M°vw = M V s * M° ST • Mtw (Equation B) 

25 Now rearranging Equation B, 

M vs = M°vw * Mtw" 1 • M^t" 1 (Equation C) 
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The calibration data calculation section 5040L 
calculates the unknown calibration data M V s according 
to Equation C and outputs the results to the 
calibration data storage section 5080L. 
5 FIG. 6 shows a process flowchart of the 

calibration apparatus according to this embodiment. 
The program code according to this flowchart is stored 
in unshown memory such as a RAM or ROM in the apparatus 
of this embodiment, and read out and executed by a CPU 

10 (not shown) . 

In Step S1040, the virtual marker picture 
generation section 5075L generates virtual pictures in 
the manner described above, and makes it displayed in 
the display section 210L. The image displayed in the 

15 display section 210L in Step S1040 is shown in FIG. 7A. 
Each X mark in the figure represents the virtual 
picture of each marker. The markers 1600 placed in 
real space is viewed by the player optically through 
the display section 210L. The image viewed by the 

20 player in the display section 210L is shown in FIG. 7B. 

In Step S1070, the virtual pictures of the markers 
and the marker images in real space are overlaid 
exactly by rotation or translation of HMD 210L (and 
thus the viewpoint 1901). (The state shown in FIG. 

25 7C.) 

When they are overlaid exactly, the operating 
section 5090L is manipulated in Step S1080. 
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In Step S1090, the calibration data calculation 
section 5040L calculates the coordinate transformation 
matrix M V s in the manner described above and outputs 
the results to the calibration data storage section 
5 5080L. 

<Derivation of the position/attitude M TW of the sensor 
coordinate system 8010 in the global coordinate system) 

The following describes how to find the 
position/attitude Mtw of the sensor coordinate system 

10 8010 in the global coordinate system 8000. 

The calibration grid 6000 is placed at a fixed 
position in real space and the magnetic sensor 220L is 
placed at a specified position (point of measurement) 
on the calibration grid 6000. There is more than one 

15 specified position. According to this embodiment, the 
output of the magnetic sensor 220L is measured at three 
points on the calibration grid 6000. The results of 
the measurements are entered in to the calibration 
section 5055L. 

20 The calibration section 5055L calculates the 

position/attitude Mtw of the sensor coordinate system 
8010 in the global coordinate system 8000, based on the 
three measurements taken with the magnetic sensor 220L, 
and stores the results in the calibration data storage 

25 section 5080. 
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The following describes in detail how the 
calibration section 5055L finds the position/attitude 
Mtw. 

FIG. 18 is a conceptual diagram illustrating how 
5 the magnetic sensor measures its position/attitude. 
Let Ow denote the origin of the global coordinate 
system; X w , Y w , and Z w the x, y, and z axes of the 
global coordinate system, respectively; O s -- the 
origin of the sensor coordinate system (expressed as 

10 coordinates in the global coordinate system); X s , Y s , 

and Z s -- the x, y, and z axes of the sensor coordinate 
system (expressed as unit direction vectors in the 
global coordinate system); P 0 , P z , Pzx -- the three 
points of measurement described above, respectively 

15 (expressed as coordinates in the global coordinate 

system); V 0 , V z , V zx -- the three points of measurement 
described above, respectively (expressed as coordinates 
in the sensor coordinate system) . The three points 
described above must satisfy the following conditions. 

20 In other words, the fixed position for the calibration 
grid 6000 and the three points of measurement on the 
grid must be set such that the following conditions 
will be satisfied. 

• The position of P Q in the global coordinate 
25 system must be known. 

* P z must be such that the vector ( P z - P Q ) is 
parallel to, and in the same direction as, Z w . 
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* P zx must be such that the plane containing the 
three points P 0 , P z , and P zx is parallel to the z-x 
plane in the global coordinate system and that the x 
component of the vector (P xz - Po) in the global 
5 coordinate system will be positive. 

Based on these conditions, the position/attitude 
Mtw of the sensor coordinate system 8010 is determined 
in the following order. 

First, since the vector (P z - P G ) is parallel to, 
10 and in the same direction as, Z w , the unit vector Z s in 
the z direction of the sensor coordinate system is 
determined according to the following equation: 

Zs = (v z -v 0 ) / | v z -v 0 | 

Next, since the vector (P zx - P Q ) is located on the 
15 z-x plane of the global coordinate system and has a 
positive x component, the unit vector Y s in the y 
direction of the sensor coordinate system is determined 
according to the following equation: 

Y s = (Zs x (v zx -v 0 ) ) / | Z s x (v zx -Vo) | 
20 Next, using Z s and Y s above, the unit vector X s in 

the x direction of the sensor coordinate system is 
determined according to the following equation: 

X s = Y s x Z s 

From Z s , Y s , and X s thus obtained, the attitude of 
25 the sensor coordinate system in the global coordinate 
system can be determined as follows. Let X s = (X x , Y x , 
Z x ), Y s = (X Y , Y Y , Z Y ), and Z s = (X z , Y z , Z z ), and the 
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attitude Rtw of the sensor coordinate system in the 
global coordinate system is given by: 
[Equation 5 ] 
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5 Next, the position of the sensor coordinate system 

in the global coordinate system (i.e., the position O s 
of the origin of the sensor coordinate system in the 
global coordinate system) is determined. The following 
method uses P 0 , but P z or P zx may be used if their 
10 global coordinates are known: 
O s = Po - Rtw _1 v 0 

Hence, the translation matrix Ttw that represents 
the position of the sensor coordinate system in the 
global coordinate system can be determined according to 
15 Equation 1. 

Consequently, the position/attitude M TW of the 
sensor coordinate system in the global coordinate 
system can be derived using the following equation. 

Mtw — R tw Ttw 

20 Based on the signals from the magnetic sensor 220L 

that takes measurements at multiple points of 
measurement, the position/attitude Mtw of the sensor 
coordinate system in the global coordinate system can 
be determined by the above method. 
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Incidentally, the position/attitude M TO of the 
sensor coordinate system in the global coordinate 
system may be determined by trial and error using 
measurements taken with a measuring tape, protractor, 
5 etc. Also, needless to say, the position/attitude of 
the AC magnetic field generator may be determined using 
any available measuring means . 
<3D image generation/presentation system) 

FIG. 8 is a drawing showing the configuration of a 
10 3D image generation/presentation system that utilizes 
the calibration data obtained through the above- 
mentioned calibration on the game device of FIG. 1. 

The image generation/presentation system outputs 
respective 3D virtual pictures (puck 1500 and goals 
15 1200 in FIG. 2) to the HMD 210L of the player 2000 and 
HMD 210R of the player 3000. The 3D virtual pictures 
for the players 2000 and 3000 are generated by the 
image generation sections 5050L and 5050R, respectively. 
This embodiment uses an Onyx2 computer system from a US 
20 company SGI for each of the image generation sections 
5050L and 5050R. 

The image generation section 5050L (5050R) 
receives puck position information and the like 
generated by a game status control section 5030 and the 
25 position/attitude of the viewpoint of the player 2000 

(3000) output from the position/attitude transformation 
section 5060L (5060R) and generates images to be 
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displayed on the HMD 210L (210R). The game status 
control section 5030 is implemented using the ONYX 2 
computer system. 

A CCD camera 230 (also shown in FIG. 1) fixed 
above the center of the table 1000 covers the entire 
surface of the table 1000 by its field of view. The 
mallet information including the mallet position 
information acquired by the camera 230 is entered into 
a mallet position measuring section 5010, which 
similarly is implemented using the ONYX 2 computer 
system from SGI . The mallet position measuring section 
5010 detects the positions of the mallets, i.e., the 
positions of the hands, of the two players. The 
information about the hand positions are entered into 
the game status control section 5030. In short, 
everything about the status and progress of the game is 
determined by the positions of the mallets. 
<Mallet position measurement > 

FIG. 9 to FIG. 11 are flowcharts showing the 
control procedures for measurement of mallet position. 

In the air hockey game, the player will never put 
his/her mallet in the other player's area. Therefore, 
to search for the mallet 260L (260 R) of the left 
player 2000 (right player 3000), the search process can 
be concentrated on the image data IL in the left field 
(image data IR) as shown in FIG. 11. The image 
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captured by the CCD camera 230 at a fixed position can 
be easily split in two as shown in FIG. 12. 

In the flowchart of FIG. 9, the mallet 260L of 
player #1 (player 2000) is searched for in Step S100 
5 and the mallet 260 R of player #2 (3000) is searched 

for in Step S200. Therefore, only the search operation 
(Step S200) for the mallet of the right player will be 
described for convenience. 

In Step S210, the above-mentioned image (multi- 

10 valued image) captured by the imaging section 230 is 
acquired. In Step S212, subroutine "Local Search" is 
run on the image data IR about the right half of the 
captured image. Details are shown in FIG. 10. If the 
coordinates (x, y) of the mallet position in an image 

15 coordinate system is found in Step S212, the flow goes 
from Step S214 to Step S220, where the coordinates (x, 
y) of the mallet position in the image coordinate 
system is transformed into the coordinate position (x' , 
y') in the coordinate system (see FIG. 14) of the table 

20 1000 according to Equation 6. 
[Equation 6 ] 
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where M T is a known 3x3 transformation matrix 
for the calibration of the image coordinate system and 
25 table coordinate system. The coordinate position (x', 
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y' ) obtained in Step S220 is sent to the game status 
control section 5030. 

If the mallet cannot be found in the local area, 
"Global Search" is run in Step S216. If the mallet is 
5 found by the "Global Search," the coordinate position 
is transformed into the position in table coordinate 
system in Step S220. The coordinate position retrieved 
by the local or global search is used for a local 
search for the mallet in the next process. 
10 FIG. 10. shows the process of a local search for 

the mallet (i.e., details of Step S212). Although the 
process shows a search in the right field for 
convenience, it can similarly be applied to search in 
the left field. 
15 In Step S220, the rectangular area made of ( 2A + 

l)x(2B+l) picture elements and defined by the 
following equations is extracted, 
x = [I'x - A, I'x + A] 
y = [I'y - A , I'y + B] 
20 where I'x and I'y are any coordinate values in the 

search area IR, and A and B are constants that 
determine the size of the search area. Such a search 
area will look like the one shown in FIG. 13. 

Step S230 is the process of extracting the picture 
25 elements whose character evaluation value IS (x, y) 
satisfies specific conditions from all the picture 
elements (x, y) in the rectangular area defined in Step 
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S220. In this process, it is preferable to use a 
methods that extracts the picture elements which are 
similar to the intensity value of the infrared rays 
emitted from the infrared emitter of the mallet. 
5 In other words, in Step S232, any picture element 

whose similarity IS exceeds a designated threshold is 
searched for. If such picture elements are found, the 
cumulative value of their generation rate is stored on 
a counter N. Also, the x coordinates and y coordinates 
10 of these picture elements are accumulated in registers 
SUMx and SUMy. Thus, 
N = N + 1 
SUMx = SUMx + x 
SUMy = SUMy + y 
15 When Step S230 is completed, the number N of all 

the picture elements in the area of FIG. 13 which are 
similar to the pattern of the infrared rays emitted 
from the mallet as well as cumulative values SUMx and 
SUMy of the coordinates are obtained. If N = 0, the 
20 result "Not Found" is output in Step S236. If N > 0, 
something that looked like a mallet was found and the 
position (Ix, Iy) of the mallet is calculated according 
to the following equations in Step S238: 
Ix = SUMx/n 
25 Iy = SUMy/n 

The mallet position (Ix, Iy) is transformed into 
coordinates in the table coordinate system. 
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FIG. 11 shows detailed procedures for the global 
search performed in Step S216. In Step S240 shown in 
FIG. 11, among the picture elements contained in the 
image IR in the right field, the maximum value of the 
5 character evaluation values IS of the picture elements 
that satisfy the following conditions is stored in a 
register Max. 

{(x, y) | x > 0, x < Width, x = nC, 
y > 0, y < Height, y = mD 
10 (where n and m are integers)} 

where C and D are constants that determine the 
roughness of the search. The definitions of width and 
height are shown in FIG. 16. In short, it is judged in 
Step S242 whether the character rating IS exceeds the 
15 threshold stored in the register Max. If any 

appropriate picture element is found, the character 
rating is set as a new threshold in Step S244 as 
follows : 

Max = IS (x, y) 
20 Ix = x 

Iy = y 

In Step S246, the coordinates (Ix, Iy) of the most 
probable picture element found in the global search is 
passed to Step S220. 
25 In this way, the mallet is found in the image, its 

coordinates are transformed into the coordinates in the 
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table coordinate system, and the results are passed to 
the game status control section 5030. 
<Game status control) 

FIG. 14 shows the game field for air hockey games 
5 according to this embodiment. This field is defined in 
the two-dimensional plane on the table 1000 and has x 
and y axes. It also has two left and right -- 
virtual goal lines 1200L and 1200R as well as virtual 
walls 1300a and 1300b installed on top and bottom in 
10 FIG. 14. The virtual goal lines 1200L and 1200R and 
virtual walls 1300a and 1300b have known coordinates 
and are not moved. In this field, the virtual image of 
the puck 1500 moves in accordance with the movement of 
the mallets 260R and 260L. 
15 The puck 1500 has coordinate information P P and 

velocity information V P of the current position, the 
left mallet 260L has coordinate information P S l and 
velocity information V S l of the current position, and 
the right mallet 260R has coordinate information P SR 
20 and velocity informat ion Vsr of the current position. 

FIG. 15 is a flowchart illustrating the processing 
procedures carried out in the game status control 
section 5030. In Step S10, the initial position P P0 and 
initial velocity V P0 of the puck 1500 are set. 
25 The puck 1500 performs uniform motion at velocity 

V P . If it hits a mallet or wall, a perfectly elastic 
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collision takes place. That is , the speed remains 
unchanged and the velocity direction is reversed. 

The game status control section 5030 obtains 
velocity information V s from the position information 
5 P s of each mallet calculated by the mallet position 
measuring section 5010. 

Step S12 is repeated at time intervals of At until 
the outcome of the game is decided (either player 
scores three points first in Step S50). 
10 Then the position of the puck 1500 is updated to 

P P = P P o + v p0 * At 

The position of the puck 1500 after the initial 
position and velocity setting are generally given by 
P P = P P + v p • At 

15 In Step S14 # it is checked whether the updated 

puck position P P is in the field of player #1 (left 
player) . The case in which the puck 1500 is located on 
the side of the left player will be described below. 
In Step S16, it is checked whether the puck is 

20 currently located such that it interferes with the 
mallet 260L of the left player. If the puck 1500 
interferes with the mallet 260L, it means that the left 
player 2000 manipulated the mallet in such a way that 
the mallet 260L collided with the puck. Thus, to 

25 reverse the motion of the puck 1500 , the sign of the x 
direction component of the puck's (1500) velocity V P is 
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changed in Step S18. Then the flow advances to Step 
S20. 

Incidentally, instead of simply changing the sign 
of the x direction component of the velocity V P , the 
puck may be made to advance in the opposite direction 
with the manipulation velocity of the mallet added as 
follows : 

Pp - ~ Ppx + VsLX 

On the other hand, if the puck currently does not 
interfere with the mallet 260L of the left player ("NO" 
in Step S16), the flow advances directly to Step S20. 

In Step S20, it is checked whether the puck is 
currently located such that it collides with the 
virtual wall 1300a or 1300b. If the answer in Step S20 
is YES, the y direction component of the puck's 
velocity V P is reversed in Step S22. 

In Step S24, it is checked whether the puck is 
currently located to the left of the goal line of the 
left player. If YES, the opposition player, i.e., the 
right (#2) player gets a score in Step S26 . In Step 
S50, it is checked whether any of the players scored 
three points or more first. If YES, the game finishes. 

If it turns out in Step S14 that the puck position 
P P is in the field of the right player (player #2), 
Step S30 and subsequent steps are performed. Steps S30 
to S40 are practically the same as Steps S16 to S26. 
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So the progress of the game is managed in the 
manner described above. The progress of the game, 
which is represented by the positions of the puck and 
mallet, is entered in the image generation sections 
5 5050 (5050L and 5050R) . 

FIG, 17 shows a^ flowchart of processes carried out 
in the position/attitude transformation sections 5060 
(5060L, 5060R) and image generation sections 5050 
(5050L, 5050R). The processes in this flowchart are 
10 common to the left and right players. 

The position/attitude transformation section 5060L 
receives the position/attitude M ST of the magnetic 
sensor in the sensor coordinate system from the 
position/attitude measuring section 5000 in Step S1701. 
15 Then, in Step S1702, it reads the calibration data M TW 
and M V s from the calibration data storage section 5080. 
In Step S1703, it determines the position/attitude M w 
of the player's viewpoint 1901 using Equation A. 
The image generation sections 5050 read the 
20 position and shape data of the goals and puck from 

memory (not shown) in the game status control section 
5030 in Step S1704, generate pictures of the goals and 
puck based on the position/attitude M w of the player's 
viewpoint 1901 in Step S1705, and output the generated 
25 pictures to the HMD 210L and HMD 210R of the respective 
players in Step S1706. 
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As described above, the calibration apparatus and 
method thereof according to this embodiment can 
calculate the positions/attitudes of the players 1 
viewpoints even if the calibration data needed to 
5 convert the output values of the magnetic sensors into 
the players' viewpoints are unknown. 
[Second Embodiment] 

In the first embodiment, the relative 
positions/attitudes M vs of the players' viewpoints as 
10 viewed from the magnetic sensors are unknown. In other 
words, both the translation component T V s and rotation 
component R V s of the transformation matrix M V s are 
unknown . 

Also, in the first embodiment, both the 
15 translation component T TO and rotation component of 
the position/attitude M TO of the sensor coordinate 
system in the global coordinate system are known. 

In relation to the second embodiment, the 
following describes how to calculate unknown 
20 calibration data, i.e., the rotation component R V s of 
M V s and translation component Ttw of M TO when the 
translation component T vs of M vs and rotation component 
Rtw of Mtw are known in some way or other. This 
embodiment has the same equipment configuration as that 
25 of the first embodiment, and differs only in the 

processing inside the calibration data calculation 
section 5040L. 
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The translation component T vs of M vs (i.e., the 
relationship between the position of the magnetic 
sensor and position of the user's viewpoint) is 
measured directly, for example, with a rule. The 
rotation component R TO of M™ is calculated using, for 
example, the method of finding R™ described in 
relation to the first embodiment. 

Alternatively, these values may be derived by 
trial and error as described in relation to M™ for the 
first embodiment, or any other measuring technique may 
be used. 

It is assumed that the known parameters are stored 
in the calibration data storage sections 5080 as is the 
case with the first embodiment. 

Separating Equation B into rotation and 
translation components , 

R°vwT°vw = RvsTvs * M° S t ' RtwTtw (Equation D) 

Re-arranging Equation D, 

Rvs'^vwT^wTtw" 1 = T V sM°stRtw (Equation E) 

Since all the terms on the right side have known 
values , if their product is calculated and replaced 
with a matrix M' and the matrix M f is decomposed into 
M' = R 1 T' (where R" is the rotation component and T' 
is the translation component), then Equation E is 
written as 



Rvs 1 R°vwT°vwTtw 1 — R ' T ' 



( Equation F ) 
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Since the rotation component and translation 
component exist as identities in the right side member 
and left side member of Equation F, respectively, R vs 
is given by: 
5 Rvs ~ R°vwR ' 1 

Similarly, T TO is given by: 

T_ rp I - ImO 
TW ~ J- J- VW 

Finally, the resulting calibration data R V s and T TO 
are output to the calibration data storage section 5080. 
10 [Third Embodiment] 

This embodiment concerns the method of calculating 
unknown calibration data, i.e. , the position/attitude 
M T w of the sensor coordinate system in the global 
coordinate system when the relative positions/attitudes 
15 of the players' viewpoints M V s as viewed from the 
magnetic sensors are known in some way or other. 

This embodiment has the same equipment 
configuration as that of the first embodiment, and 
differs only in the processing inside the calibration 
20 data calculation section 5040L. The unknown 

calibration data M™ is calculated by rearranging 
Equation B as follows, and is output to the calibration 
data storage section 5080L. 
M™ = M°st 1 * M V s 1 * M°vw 
2 5 Alternatively, M V s may be derived by the method of 

the first embodiment; by trial and error based on 
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values measured with a rule, protractor, etc.; or by 
any other measuring means. 
[ Fourth Embodiment ] 

Although the first to third embodiments relate to 
5 calibration of magnetic sensors which are 
position/attitude sensors for measuring the 
positions/attitudes of objects, the present invention 
can also be applied to calibration of attitude sensors 
that measure only the attitudes of objects. The fourth 
10 embodiment concerns the method of determining the 

calibration data needed to convert the output values of 
attitude sensors (i.e., the attitudes of the attitude 
sensors in the sensor coordinate system) into attitudes 
of the players' viewpoints in the global coordinate 
15 system for a game device that uses attitude sensors. 

This embodiment has the same equipment configuration as 
that of the first embodiment except that it comprises 
attitude sensors and an attitude measuring section. 
Now let Rtw denote the attitude of the sensor 
20 coordinate system in the global coordinate system, R S t 
the attitude of the attitude sensor (output values 
from attitude sensor) in the sensor coordinate system, 
and R V s the relative attitude of the player's 
viewpoint as viewed from the attitude sensor. 
25 Incidentally, although most attitude sensors do not 

have explicit devices for defining a sensor coordinate 
system, such as the AC magnetic field generator in the 
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case of magnetic sensors, if it is assumed that the 
sensor attitude in the global coordinate system 
corresponds to R™ when sensor output is pointing to 
the origin (i.e., when R ST = I is output), the 
5 coordinate transformation is given by a relation 

similar to those used in the first to third embodiments. 
Thus, the Equation A can be expressed as 

Rvw = Rvs * Rst * Rtw (Equation G) 

where R ST is the input into the position/attitude 
10 transformation section 5060L, Rvw is the output from 

the position/attitude transformation section 5060L, and 
Rvs and R TW are the calibration data needed to convert 
Rst into Rvw- Thus, the position/attitude 
transformation section 5060L calculates Rvw according 
15 to Equation G using the input R ST from the 

position/attitude measuring section 5000 as well as R V s 
and Rtw stored in the calibration data storage section 
5080L. The position/attitude transformation section 
5060L further calculates the position/attitude Mvw of 
20 the player's viewpoint in the global coordinate system, 
based on the calculated attitude Rvw of the player's 
viewpoint in the global coordinate system and the 
position Tvw of the player's viewpoint in the global 
coordinate system, and outputs the results to the image 
25 generation section 5050L. (Tvw may be the values 

measured by a position sensor such as an ultrasonic 
sensor or optical sensor. Alternatively, predetermined 
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fixed values may be used if the position of the 
viewpoint does not move or the amount of travel is 
negligible. Also, position information obtained by any 
other means may be used. ) 



the sensor coordinate system, only the rotation 
component Ry T w around the y axis (i.e., in the azimuth 
direction) is unknown and that the rotation component 
Rxtw around the x axis and rotation component Rztw 

10 around the z axis are unit matrices (i.e., R™ = Ry™) • 
Since many attitude sensors can output values in a 
coordinate system in real space by measuring the 
direction of the earth gravitation except values in the 
azimuth direction (around the y axis, in the yaw 

15 direction), the y axes of the global coordinate system 
and sensor coordinate system coincide if the global 
coordinate system is defined as being parallel to the 
ground surface. Thus, Rx TO and Rz^ can be considered 
to be unit matrices. 

20 This embodiment also assumes that, of the relative 

attitude R V s of the player's viewpoint as viewed from 
the sensor, the rotation component Ry sv around the y 
axis of its inverse matrix R sv (relative attitude of 
the attitude sensor as viewed from the player's 

25 viewpoint) is known in some way or other while the 

rotation component Rx sv around the x axis and rotation 
component Rz sv around the z axis are unknown. Ry sv may 



5 



This embodiment assumes that of the attitude R™ of 
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be derived by trial and error using values measured 
with a protractor, etc. , or by any other measuring 
means . 

The known data Rxtw, Rz™, and Ry S v have been stored 
in the calibration data storage section 5080L. 

Calibration is performed by moving the attitude Rvw 
of the viewpoint 1901 of the player wearing the HMD 
210L to a predetermined attitude R°vw (= Ry°vw) and 
acquiring the sensor output R° ST at that time. Using 
Equation G, the relationship among these data can be 
expressed as 

Ry°vw = (RzsvRxsvRysv) _1 R°sTRyTw (Equation H) 

Rearranging Equation H 

Rz sv RxsvRysvRy°vw = Rz°sTRx 0 s T Ry 0 sTRyTw 

Since both left -side and right-side members are 
products of rotation components around the z, x, and y 
axes, an identity holds for the rotation component 
around each axis: the z, x, and y axes. The identities 
of the rotation components around the z and x axes are 
given as 

Rzsv = Rz°st 

Rxsv ~ Rx°sx 

from which Rz S v and Rx sv can be determined. 

On the other hand, the identity of the rotation 
component around the y axis can be determined from the 
following equation . 

Ry S vRy°vw = Ry 0 ST RyTw 
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Thus , Ry TW is given as 



RYtw - RysvRy°vwRy°sT 



-i 



The calibration data calculation section 5040L 



according to this embodiment calculates the calibra 




5 data Rz sv , Rx sv , and Ry*rw as described above, further 

calculates R vs (= (Rzsv Rx sv Rysv)~l) and Rtw (= Rytw) , and 
then outputs the results to the calibration data 
storage section 5080L. 



10 retains the predetermined position T°vw of the viewpoint 
needed for the virtual marker picture generation 
section 5075L to generate virtual pictures of markers 
in addition to the predetermined attitude R°vw of the 
viewpoint needed to calculate the calibration data 
15 described above. 
<Variation 1> 

Although the embodiments described above use an 
optical HMD, the present invention is not limited to 
optical HMDs, but it can also be applied to video see- 
20 through type HMDs. 

In that case, the measuring object, which is the 
player's viewpoint 1901 according to the embodiments 
described above, will be an imaging apparatus (e.g., a 
video camera) installed in the video see-through type 
2 5 HMD. The predetermined position/attitude M°vw will be 
given as the position/attitude of the viewpoint of the 
imaging apparatus and the transformation matrix M V s 



The memory 504 5L according to this embodiment 
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will be given as the transformation matrix that 
represents the position/attitude of the imaging 
apparatus as viewed from the magnetic sensor 220L. 

According to this variation, the virtual marker 
5 picture generation section 507 5L superimposes a virtual 
picture of markers over the picture (marker image) of 
real space captured by the imaging apparatus , based on 
the predetermined position/attitude M°vw, and displays 
the resulting picture in the display section. The 

10 markers 1600 must be such that they can be 

distinguished from the table 1000 in the marker image. 
The operator manipulates the position/attitude of the 
imaging section to overlay the real and virtual 
pictures of the markers. In video see -through mode, 

15 the display section does not always need to use the HMD 
210L of the game device during calibration unlike in 
optical see- through mode, but a CRT display or other 
display units may be used. 
< Variation 2> 

20 Even if a game device uses an optical HMD, 

calibration may be performed in video see-through mode 
as with Variation 1 if the HMD 210L is equipped with an 
imaging apparatus whose relative position/attitude as 
viewed from the player's viewpoint is known. 

25 According to this variation, the virtual marker 

picture generation section 5075L superimposes a virtual 
picture of markers over the picture (marker image) of 
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real space captured by the imaging apparatus , according 
to the position/attitude of the imaging apparatus in 
the global coordinate system, determined by a 
predetermined position/attitude M°vw of the player's 
5 viewpoint and the position/attitude of the imaging 
apparatus as viewed from the player's viewpoint, and 
displays the resulting picture in the display section. 
If the HMD 210L is used as the display section, it is 
desirable to make display surface light-tight by 

10 adjusting its transmittance . 
<Variation 3> 

Although the above embodiments use the markers 
1600 -- which are point information as a guide in 
alignment, any geometry information may be used as long 

15 as it provide visual clues when moving the 

position/attitude of a viewpoint to a predetermined 
position/attitude M°vw- For example, the user may 
capture the geometry of a real object such as a desk 
from the environment and draw its wire -frame model 

20 according to a predetermined position/attitude M°vw- In 
this case, it is possible to lead the viewpoint to the 
position/attitude M°vw by moving the viewpoint so as to 
match the image of the real desk and the superimposed 
picture of the wire-frame model. The viewpoint can 

25 also be led to the position/attitude M°vw as follows: if 
the virtual object (e.g., the virtual puck in the first 
embodiment) displayed after calibration is drawn 
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according to the predetermined position/attitude M°vw, 
the viewpoint can be moved so as to place the virtual 
puck at the appropriate position on the real desk. 
Incidentally, it goes without saying that some of the 
5 above-mentioned approaches can be used in combination. 
<Variation 4> 

Although the above embodiments use the calibration 
grid 6000 to help place the viewpoint in the 
position/attitude M°vw, it is not always necessary to 
10 use a calibration grid: visual information alone can be 
used as a guide. It is also possible to place the 
viewpoint in the position/attitude M°vw by placing the 
HMD 210L in an established attitude at an established 
position on the calibration grid 6000 instead of using 
15 visual information. Or any other method may be used as 
long as it leads the viewpoint to a predetermined 
position/attitude (or attitude). 
<Variation 5> 

Although the above embodiments have been applied 
20 to air hockey games, it goes without saying that they 
can be applied to systems that present mixed reality 
other than air hockey games. Besides, it is not that 
they are suitable only for coordinated work by two 
persons. They are also applicable to systems that 
25 present mixed reality to one or more than two workers 
(or players). They are not limited to systems that 
present mixed reality, but they can be used for any 
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application that measure the position/attitude of the 
viewpoint of an imaging apparatus by means of magnetic 
sensors. Furthermore, they are not limited to the 
measurement of the position/attitude of the player's 
viewpoint described in relation to variation 2, but 
they can be used for applications that measure the 
position/attitude of any object whose position/attitude 
relative to the viewpoint of the imaging apparatus is 
known . 

<Variation 6> 

Although the above embodiments use magnetic 
sensors that employ AC magnetic fields, they are not 
limited to such magnetic sensors, but they can be 
applied to gyro-sensors, ultrasonic sensors, or any 
other sensors that can detect the position/attitude or 
attitude of a measuring object. 

Note that the present invention may be applied to 
either a system constituted by a plurality of devices 
(e.g., a host computer, an interface device, a reader, 
a printer, and the like), or an apparatus consisting of 
a single equipment (e.g., a copying machine, a 
facsimile apparatus, or the like). 

The objects of the present invention are also 
achieved by supplying a storage medium, which records a 
program code of a software program that can implement 
the functions of the above-mentioned embodiments to the 
system or apparatus, and reading out and executing the 
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program code stored in the storage medium by a computer 
(or a CPU or MPU) of the system or apparatus. 

In this case, the program code itself read out 
from the storage medium implements the functions of the 
5 above-mentioned embodiments, and the storage medium 
which stores the program code constitutes the present 
invention . 

As the storage medium for supplying the program 
code, for example, a floppy disk, hard disk, optical 
10 disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, 
nonvolatile memory card, ROM, and the like may be used. 

The functions of the above-mentioned embodiments 
may be implemented not only by executing the readout 
program code by the computer but also by some or all of 
15 actual processing operations executed by an OS 

(operating system) running on the computer on the basis 
of an instruction of the program code. 

The present invention includes a product, e.g., a 
printout, obtained by the image processing method of 
20 the present invention. 

Furthermore, the present invention also includes a 
case where, after the program codes read from the 
storage medium are written in a function expansion card 
which is inserted into the computer or in a memory 
25 provided in a function expansion unit which is 

connected to the computer, CPU or the like contained in 
the function expansion card or unit performs a part or 
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entire process in accordance with designations of the 
program codes and realizes functions of the above 
embodiments . 

In a case where the present invention is applied 
5 to the aforesaid storage medium, the storage medium 
stores program codes corresponding to the flowcharts 
(FIGS. 6 and/or FIGS. 9 and/or FIGS. 10 and/or FIGS. 11 
and/or FIGS. 15 and/or FIGS. 17) described in the 
embodiments . 

10 As many apparently widely different embodiments of 

the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 

15 appended claims. 
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